{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Autograd in MinPy\n",
    "\n",
    "This tutorial is also available in step-by-step notebook version on [github](https://github.com/dmlc/minpy/blob/master/examples/tutorials/autograd_tutorial.ipynb). Please try it out!\n",
    "\n",
    "Writing backprop is often the most tedious and error prone part of a deep net implementation. In fact, the feature of autograd has wide applications and goes beyond the domain of deep learning. MinPy's autograd applies to any NumPy code that is imperatively programmed. Moreover, it is seemlessly integrated with MXNet's symbolic program (see [for example](../tutorial/complete_sol_opt_guide/complete.rst)). By using MXNet's execution engine, all operations can be executed in GPU if available.\n",
    "\n",
    "## A Close Look at Autograd System\n",
    "MinPy's implementation of autograd is insprired from the [Autograd project](https://github.com/HIPS/autograd). It computes a gradient function for any single-output function. For example, we define a simple function `foo`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "16"
      ]
     },
     "execution_count": 1,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "def foo(x):\n",
    "    return x**2\n",
    "\n",
    "foo(4)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we want to get its derivative. To do so, simply import `grad` from `minpy.core`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "import minpy.numpy as np  # currently need import this at the same time\n",
    "from minpy.core import grad\n",
    "\n",
    "d_foo = grad(foo)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "8.0"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "d_foo(4)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can also differentiate as many times as you want:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "d_2_foo = grad(d_foo)\n",
    "d_3_foo = grad(d_2_foo)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now import `matplotlib` to visualize the derivatives."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAlsAAAHfCAYAAABnDB0iAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzt3XeY1NXZh/H7CPbeiRVRY0k0scQadMWKxhILlsReorEQ\nY2JPIEaNaOwdUMSCCmrAYAMUxI6CiCIqaqwI+kpU0KgI5/3jLIpI29mZOVPuz3XNtbuzOzNfdFme\nPef5PSfEGJEkSVJpzJc7gCRJUi2z2JIkSSohiy1JkqQSstiSJEkqIYstSZKkErLYkiRJKqF5LrZC\nCDeEECaEEEbNcN/SIYQBIYRXQwgPhRCWnOFzZ4QQxoYQxoQQdip2cEmSpGrQlJWtHsDOM913OjAo\nxrgO8AhwBkAIYX2gA7Ae0B64JoQQmh9XkiSpusxzsRVjfBz470x37wn0bHy/J7BX4/t7AHfEGL+J\nMb4FjAU2a15USZKk6tPcnq0VYowTAGKM44EVGu9fGXh3hq97v/E+SZKkutKyyM/X5LN/QgieFyRJ\nkqpGjLFJrVHNLbYmhBBWjDFOCCG0Aj5svP99YNUZvm6Vxvtm6cEHIzvP3A0mzULnzp3p3Llz7hiq\nAn6vqCn8ftG8OOQQuOWWpregN3UbMTTeprsXOKzx/UOBfjPcf0AIYYEQwhrAWsCw2T1pjx5NTCFJ\nklRGn30G995b2GObMvqhF/Ak8OMQwjshhMOBC4AdQwivAts3fkyM8WWgN/AycD/w+xjjbLcLH3wQ\nJk4s7A8gSZJUanfeCe3aFfbYplyNeFCMcaUY44IxxtVijD1ijP+NMe4QY1wnxrhTjPGTGb7+HzHG\ntWKM68UYB8zpudu3h9tvL+wPoPrS0NCQO4KqhN8ragq/XzQ3PXrAEUcU9tgwhwWnsgghxIceipxx\nBgwfnjWKJEnSD4wZk1a13n0X5p8/NLlBviKO69l+e/joI3jhhdxJJEmSvu+mm1JzfMsCLyusiGKr\nRQs49FAb5SVJUmX55hu4+WY4/PDCn6Miii2Aww6DXr3g669zJ5EkSUoefBBat4Z11y38OSqm2Fpz\nTVh/ffj3v3MnkSRJSm68sfDG+OkqokF+eoabb4bevaF//6yRJEmS+OgjWHtteOcdWGKJdF8IVdog\nP90++8ATT8C4cbmTSJKkenfrrbDHHt8VWoWqqGJr0UVh333hlltyJ5EkSfUsxnThXnMa46erqGIL\n0h/qxhvTH1KSJCmH4cNh8mTYdtvmP1fFFVtbbgkhwFNP5U4iSZLqVY8eaVLCfEWolCqqQX66Ll1g\n7Fjo3j1TKEmSVLe+/BJWXhlGjIDVV//+56q+QX66gw+Gu++Gzz/PnUSSJNWbvn1h441/WGgVqiKL\nrZVWgq23hrvuyp1EkiTVm2LM1ppRRRZbkBrlPb5HkiSV0zvvpOb4vfYq3nNWbLG1++7w8svwxhu5\nk0iSpHrRsyfsvz8svHDxnrNii60FFoCDDkonbUuSJJXatGmp7ijGbK0ZVWyxBekPe9NNMHVq7iSS\nJKnWDR0KiywCm25a3Oet6GLrZz+DFVaAQYNyJ5EkSbXuxhvTQk9o0mCHuavIOVszuvZaeOQR6NOn\njKEkSVJd+eQTaN06zflcfvnZf13NzNma0UEHwcCB8OGHuZNIkqRaddttsPPOcy60ClXxxdaSS6bL\nL2++OXcSSZJUi2KEbt3gqKNK8/wVX2wBHH10OrrHw6klSVKxDR8On34K229fmuevimJrq63SQZCP\nP547iSRJqjXdu8ORRxbn0OlZqfgG+ekuuQRGjnQ7UZIkFc/kybDaavDii+nw6bmpyQb56Q4+GO69\nN10tIEmSVAx9+qTzmOel0CpU1RRbyy+frhK47bbcSSRJUq3o1i31hpdS1RRbkP5jdOtmo7wkSWq+\n0aPhrbdg111L+zpVVWy1aweffZauGpAkSWqO7t3TxPiWLUv7OlXTID/deefBO+/A9deXMJQkSapp\nX34Jq64KzzwDbdrM++NqukF+usMOg96909UDkiRJhejbN53B3JRCq1BVV2ytvDK0bZsKLkmSpEKU\nozF+uqortiCN0+/ePXcKSZJUjd54A0aNSscBlkNVFlu77pquHhg9OncSSZJUbW64Ic3vXHDB8rxe\n1TXIT3fWWfDFF3DppSUIJUmSatI336SJ8YMGwfrrN/3xddEgP92RR8Ktt6arCSRJkubFfffBGmsU\nVmgVqmqLrTZt0lUEffvmTiJJkqpF9+6p97ucqrbYgu8mykuSJM3Ne+/BE09Ahw7lfd2qLrb22itd\nTfDGG7mTSJKkSnfTTbD//rDoouV93aouthZcMF1NcMMNuZNIkqRKNm1aqhfKvYUIVV5sQdpKvPFG\n+Prr3EkkSVKlGjAAll4aNtmk/K9d9cXWeuvBuutCv365k0iSpEp13XVw3HF5Xrtq52zN6I47UqP8\nww8XKZQkSaoZ770HG24I77wDiy3WvOeqqzlbM/r1r+Gll+C113InkSRJleaGG+DAA5tfaBWqJla2\nAE47DaZOhX/+swihJElSTfjmG2jdGu6/P61uNVfdrmwBHHMM9OzpRHlJkvSd++5Lx/MUo9AqVM0U\nW2uuCRtvDHfdlTuJJEmqFNddB8cemzdDzRRbkP5jXndd7hSSJKkSvPkmPPss7Ldf3hw1VWz96lfw\nn/+kZnlJklTfunWDQw6BhRfOm6NmGuSn69QJJk6EK68s2lNKkqQq8/XXqVfr0UdhnXWK97x13SA/\n3VFHwW23weef504iSZJy6ds3DT4vZqFVqJortlZdFX75yzToVJIk1adKaIyfruaKLbBRXpKkevbK\nKzB6dBp6XglqstjaeWf46CN47rncSSRJUrl17QpHHAELLJA7SVJzDfLTnX9+ujKxW7eiP7UkSapQ\n//tfaikaNgzatCn+89sgP4MjjkgDTj/9NHcSSZJULnfdBb/4RWkKrULVbLHVqhXssEO6MlGSJNWH\n666D3/0ud4rvq9liC75rlM+8UypJkspg1Ch4++005LyS1HSxtd126WDqp57KnUSSJJXa9deneZst\nW+ZO8n012yA/3cUXwwsvwM03l+wlJElSZpMnp4nxo0bBKquU7nUKaZCvsNqv+A47DNZcEz7+GJZd\nNncaSZJUCrffDm3blrbQKlRNbyNCKrD23BNuvDF3EkmSVAoxwtVXw/HH504yazVfbEH6j3/NNTB1\nau4kkiSp2J54Ar74Ik0hqER1UWxtthkstxw88EDuJJIkqdiuvhp+/3uYr0KrmppvkJ+uZ890OLUF\nlyRJtWP8eFhvvXRqzFJLlf71nCA/B/vvD8OHw9ixuZNIkqRi6dYNOnQoT6FVqLpZ2QI47TSYMgUu\nuaQsLydJkkpoyhRo3TrtWm24YXleM9vKVgjh5BDCSyGEUSGE20IIC4QQlg4hDAghvBpCeCiEsGQx\nXqs5jjsuzdv6/PPcSSRJUnP165fGO5Wr0CpUs4utEMJKwInAxjHGDUmzuw4ETgcGxRjXAR4Bzmju\nazVX69aw1VbQq1fuJJIkqbkqedzDjIrVs9UCWDSE0BJYGHgf2BPo2fj5nsBeRXqtZjn++PQ/x/MS\nJUmqXqNHw6uvwq9/nTvJ3DW72IoxjgMuBt4hFVmfxhgHASvGGCc0fs14YIXmvlYx7Lhj2kZ88snc\nSSRJUqGuvhqOPhoWWCB3krlr9nE9IYSlSKtYqwOfAn1CCL8BZl47mu1aUufOnb99v6GhgYaGhubG\nmq355vtudWvrrUv2MpIkqUQ++yyNc3rppdK/1pAhQxgyZEiznqPZVyOGEPYFdo4xHt348cHAFkA7\noCHGOCGE0AoYHGNcbxaPL9vViNN98gmssQaMGQOtWpX1pSVJUjNddRUMHQq9e5f/tXNdjfgOsEUI\nYaEQQgC2B14G7gUOa/yaQ4F+RXitolhqKdhvvzSbQ5IkVY9KPwdxVooyZyuE0Ak4AJgCPA8cBSwO\n9AZWBd4GOsQYP5nFY8u+sgXwwguw225p4uz885f95SVJUgEefhj+8AcYNQpCk9aXiqOQla26Gmo6\ns7ZtoWNH2HffLC8vSZKaaO+9Yaed4Nhj87y+xVYT3XEHXH89DB6c5eUlSVITvPMObLQRvP02LLZY\nngyejdhEe++dZnSMHp07iSRJmpvrr4ff/jZfoVWoul7ZAujUCT76CK65JlsESZI0F199BauvDo8+\nCuusky+HK1sFOOYYuP32NLNDkiRVprvugg02yFtoFarui62VV05T5Xv2nPvXSpKkPKpt3MOM6r7Y\nAjjxRLjySpg2LXcSSZI0s2HD4IMPYPfdcycpjMUW8Mtfpma7Bx7InUSSJM3s8svhhBOgRYvcSQpT\n9w3y0/XsCbfdBgMG5E4iSZKmGzcOfvpTePPNdAJMbjbIN8MBB6RptC+/nDuJJEma7tpr4cADK6PQ\nKpQrWzPo3BnGj4frrsudRJIkffllGvcwdGjlXIXoBPlmGj8e1lsP3ngDllkmdxpJkupbjx7Qu3dl\n9VS7jdhMrVqlKx26d8+dRJKk+hZjaozv2DF3kuaz2JpJx45plsc33+ROIklS/Ro6NE2N32mn3Ema\nz2JrJptsAquuCn375k4iSVL9uvxyOOkkmK8GKhV7tmahTx+44gp47LHcSSRJqj9vvQWbbgpvvw2L\nLpo7zffZs1Ukv/51+h88YkTuJJIk1Z+rroLDD6+8QqtQrmzNRpcuaeaWZyZKklQ+kydD69YwfHga\n+1BpClnZalmqMNXu6KNhzTVhwgRYccXcaSRJqg833wzbbluZhVah3EacjWWWgf33d8CpJEnlMm1a\n6pmuhXEPM7LYmoOTTkrHBHz1Ve4kkiTVvocegoUXhrZtcycpLoutOVh/fdhwQ7jzztxJJEmqfdOH\nmIYmdURVPoutuejYMf3Pr8AefkmSasYrr8DIkXDAAbmTFJ/F1ly0bw+TJsETT+ROIklS7briCjjm\nGFhoodxJis/RD/PgyivTsQF9+uROIklS7fnvf6FNmzRy6Uc/yp1mzgoZ/WCxNQ8mTYI11oDnnkuz\nPyRJUvFceCG8+CLcckvuJHNnsVVCp54KU6bApZfmTiJJUu34+uu0qtW/P/z857nTzJ3FVgm99166\nMvHNN2GppXKnkSSpNtxyC9x0Ezz8cO4k88azEUtolVVg112ha9fcSSRJqg0xwsUXwymn5E5SWhZb\nTXDKKelqia+/zp1EkqTq98gj6d/UXXbJnaS0LLaaYKON4Mc/ht69cyeRJKn6XXwx/PGPMF+NVyP2\nbDXRfffB2WfDiBG1N+FWkqRyefllaNcO3nqrumZr2bNVBu3bp7MSBw/OnUSSpOp1ySVw/PHVVWgV\nypWtAnTvDv/6V1rlkiRJTTNhAqy3Hrz2Giy3XO40TePKVpn89rcwfDiMGZM7iSRJ1efqq2H//auv\n0CqUK1sFOuccePdd6NYtdxJJkqrHF1+k01gefzxddFZtHGpaRh99lL5JXnkFVlwxdxpJkqrDtdfC\ngw9Cv365kxTGbcQyWn75tAR69dW5k0iSVB2mTUvH3tX6ENOZWWw1w8knw3XXpSVRSZI0Z//+Nyy5\nJLRtmztJeVlsNcM668AWW8DNN+dOIklS5Zt+NE+9zam0Z6uZHn0UjjkmXZlY6xNwJUkq1LPPwr77\nwhtvQMuWudMUzp6tDLbZBhZfHPr3z51EkqTKdfHF0LFjdRdahXJlqwhuvz31bj36aO4kkiRVnrff\nho03hv/8B5ZYInea5nFlK5N9903fQM8+mzuJJEmV5/LL4fDDq7/QKpQrW0Vy2WXwxBPQp0/uJJIk\nVY6JE2GtteCFF2DVVXOnaT6HmmY0eTKssQY8+SSsvXbuNJIkVYa//x3efBN69MidpDgstjL7619h\n/Hjo2jV3EkmS8vvii7QQMWRIOni6FlhsZfbRR2n21ujR8KMf5U4jSVJeV18NAwdC3765kxSPxVYF\nOPFEWGQR6NIldxJJkvL55pvUVtOrF2y5Ze40xWOxVQHeegs22STtTy+5ZO40kiTl0asXXH997Y1F\ncvRDBWjdGtq3T3O3JEmqRzGmHZ7TTsudpDJYbJXAqaemURBffpk7iSRJ5ffgg6ngat8+d5LKYLFV\nAhtumCblekC1JKkeTV/VqrcDp2fHnq0SGToUjjwSXnkFWrTInUaSpPJ4+mk48EAYO7Y2z0G0Z6uC\ntG0Lyy0H99yTO4kkSeXTpQucckptFlqFcmWrhPr1S5Nzn33WpVRJUu0bMwYaGtJ5wYsskjtNabiy\nVWF23z1Nz33kkdxJJEkqvYsughNOqN1Cq1CubJXYTTfBbbelCbqSJNWq995LF4i9/joss0zuNKXj\nylYFOuig1CQ/fHjuJJIklc6ll8Jhh9V2oVUoV7bK4JJL4Jln4M47cyeRJKn4Jk6EtdaCF16AVVfN\nnaa0PK6nQk2aBG3awFNPpW9GSZJqybnnwhtvQI8euZOUnsVWBfvLX+DDD9M5UZIk1YovvkgLCoMH\nw3rr5U5TehZbFeyjj2CddeDFF2HllXOnkSSpOK64IhVa//pX7iTlYbFV4U45BaZNS02EkiRVu6++\ngjXXTHMlN9kkd5rysNiqcOPGwU9/mq5OXGGF3GkkSWqerl2hb1+4//7cScrHYqsKHH88LLEE/OMf\nuZNIklS4KVNSe8ytt8JWW+VOUz7Z5myFEJYMIfQJIYwJIYwOIWweQlg6hDAghPBqCOGhEMKSxXit\nanfqqek3gYkTcyeRJKlwt98OrVvXV6FVqGINNb0cuD/GuB7wM+AV4HRgUIxxHeAR4IwivVZVW311\n2Guv1FAoSVI1mjoVzj8fzj47d5Lq0OxtxBDCEsDzMcY1Z7r/FWDbGOOEEEIrYEiMcd1ZPL6uthEB\nxo6FLbeEN99MW4qSJFWT3r3hssvgiScgNGlDrfrl2kZcA/i/EEKPEMKIEELXEMIiwIoxxgkAMcbx\ngC3hjdZeG3beGa65JncSSZKaZtq0NMT07LPrr9AqVMsiPcfGwPExxudCCJeSthBnXq6a7fJV586d\nv32/oaGBhoaGIsSqbGeeCe3awYknwqKL5k4jSdK86d8fWraE9u1zJymPIUOGMGTIkGY9RzG2EVcE\nnooxtmn8+JekYmtNoGGGbcTBjT1dMz++7rYRp9tnH2jbFv7wh9xJJEmauxhh883h9NNh771zp8kj\nyzZi41bhuyGEHzfetT0wGrgXOKzxvkOBfs19rVpz1llw0UXw5Ze5k0iSNHcDB8Lnn6cLvTTvijJn\nK4TwM6A7MD/wJnA40ALoDawKvA10iDF+MovH1u3KFsBuu8Huu8Oxx+ZOIknSnG2zTfr36qCDcifJ\nx6GmVeipp+DAA9MVivPPnzuNJEmzNnQoHHkkjBmTerbqVbahpircllumc6Vuuy13EkmSZu/cc+GM\nM+q70CqUK1sVYPBg+N3v0m8LLVrkTiNJ0vc98wx06JB2YRZYIHeavFzZqlINDbD88tCnT+4kkiT9\n0HnnwWmnWWgVypWtCvHAA/DnP8OoUTCfJbAkqUKMHAm77ppOPVloodxp8nNlq4rtskv6Jr7nntxJ\nJEn6zjnnwJ/+ZKHVHK5sVZD+/VPz4QsvuLolScpv+qrW66/DIovkTlMZXNmqcrvtBgsvDHfdlTuJ\nJEnQuTOceqqFVnO5slVhHnggLdeOGuWViZKkfEaMSEO3X389LQQocWWrBuyyCyy+uFcmSpLy6tw5\nXYFoodV8rmxVoIceSodTv/SSq1uSpPJ77rl0/uHrr9sYPzNXtmrETjvB0kvDnXfmTiJJqkedO6cL\ntiy0isOVrQo1cCCceCKMHu3qliSpfIYNg332SataCy6YO03lcWWrhuywAyy3HNx+e+4kkqR60rkz\nnHmmhVYxubJVwR5+GI47Dl5+2YM/JUml9/TTsP/+8NprFluz48pWjWnXDn70I+jVK3cSSVI96NTJ\nVa1ScGWrwg0ZAkcfDWPGuLolSSqdJ5+Egw5Kq1oeOD17rmzVoIYGWGUVuPXW3EkkSbWsUyc46ywL\nrVJwZasKDB0Khx8Or7wC88+fO40kqdY8/jgcfHBa1fLfmTlzZatGbbMNtG4Nt9ySO4kkqRZ16gRn\nn22hVSqubFWJxx+HQw6BV1/1L4MkqXjcPWkaV7Zq2C9/CWuuCTfdlDuJJKlWxOiqVjm4slVFnn4a\nOnRIe+oeoSBJaq6BA+H4453n2BSubNW4LbaAjTaC667LnUSSVO1iTDO1/v53C61Ss9iqMueeCxdc\nAJMm5U4iSapm//oXfPMN7Ldf7iS1z2KrymywQTo38bLLcieRJFWrqVNTn9Z558F8VgIlZ89WFXrj\nDdh883Rl4rLL5k4jSao2PXtC9+7pSsTQpO4jFdKzZbFVpY47DhZbDC66KHcSSVI1+eorWHdduPlm\naNs2d5rqY7FVR8aNS1uKo0bByivnTiNJqhZXXQX3359uajqLrTpz6qnw2WdenShJmjeffw5rrZUK\nrY02yp2mOlls1ZmPP4Z11knzt9ZaK3caSVKl+8c/YORIuPPO3Emql8VWHfr739MRC7fdljuJJKmS\n/fe/8OMfwxNPpLcqjMVWHZo0CdZeGwYMgA03zJ1GklSpzjwTPvwwXYWowlls1anLLoNHHoF7782d\nRJJUicaPh5/8JG0hrrpq7jTVzWKrTn35ZVoSvvNO2HLL3GkkSZXmxBPTkTyXXpo7SfWz2KpjN9wA\nt96aVrgcUCdJmu6tt2CTTVJ/7/LL505T/TyIuo4demiavTVwYO4kkqRK0rkzHH+8hVZOrmzVkN69\n4cILYdgwz7qSJMHLL0NDA4wdC0sumTtNbXBlq87tu29627t33hySpMpw+ulw2mkWWrm5slVjBg+G\nI4+EMWNgwQVzp5Ek5fLoo3DYYalXy38PiseVLbHddrD++nDNNbmTSJJymTYN/vxnOP98C61KYLFV\ng7p0SUcy/Pe/uZNIknLo0ycVXPvvnzuJwG3EmnX00bDMMqnwkiTVj6++gvXWSyOBttsud5ra45wt\nfWvcONhgAxgxAlZfPXcaSVK5XHYZDBoE/fvnTlKbLLb0PX/9axpmd/PNuZNIksrhk0/SiSKDB6fj\neVR8Flv6nkmT0l+6+++HjTbKnUaSVGqnnQYTJ0K3brmT1C6LLf3AtdfC3XenyfIe4yNJtevtt2Hj\njeHFF2GllXKnqV2OftAPHHUUvPsuPPRQ7iSSpFL6y1/SsTwWWpXHla060Ldv6t96/nlo0SJ3GklS\nsT3/PLRvn47lWXzx3GlqmytbmqU994QllrBRXpJqUYxpgOlf/2qhValc2aoTTz+dzk587TVYZJHc\naSRJxfLgg9CxI7z0Esw/f+40tc+VLc3WFlvAllum+SuSpNowdSqceipccIGFViVzZauOvP56KrrG\njIHll8+dRpLUXD16pEnxjz3mFefl4ugHzVXHjjBligdVS1K1mzwZ1l03nYO45Za509QPiy3N1cSJ\n6S/nww+n43wkSdXpr39NOxa9euVOUl8stjRPrroqjYNw0KkkVad33kkngzz/PKy2Wu409cUGec2T\nY4+FDz6Ae+/NnUSSVIgzzkgDTC20qoMrW3VqwAD4/e9h9GhYcMHcaSRJ8+qpp2C//eCVV2CxxXKn\nqT+ubGme7bQTrLceXHFF7iSSpHk1bRqcfDKcf76FVjVxZauOvfYabLVVWt1accXcaSRJc3PbbWle\n4jPPwHwul2Rhg7ya7JRT4LPPoFu33EkkSXPy+efpavI77oCtt86dpn5ZbKnJPvkk/eV94IF0ZYsk\nqTL97W9pKPUdd+ROUt8stlSQ669Pc1qGDHEUhCRVovfeg5/9DEaMgNVXz52mvtkgr4IcdVRa4br7\n7txJJEmzcuqpcNxxFlrVypUtATB4MBxxRFqiXmih3GkkSdM99hj85jfp5/Oii+ZOI1e2VLDttks9\nW5dckjuJJGm6qVPhxBPhoosstKpZ0YqtEMJ8IYQRIYR7Gz9eOoQwIITwagjhoRDCksV6LZXGP/+Z\niq1x43InkSQBdO0KSy0FHTrkTqLmKObKVkfg5Rk+Ph0YFGNcB3gEOKOIr6USaNMm9W+d4f8pScru\n44+hU6c0fNqLl6pbUXq2QgirAD2A84A/xhj3CCG8AmwbY5wQQmgFDIkxrjuLx9qzVUEmTUqT5fv0\ngS23zJ1GkurX8cenIuuqq3In0YwK6dlqWaTXvhT4MzDjVuGKMcYJADHG8SGEFYr0WiqhxReHCy9M\nf8mffRZatMidSJLqzwsvwF13paZ4Vb9mbyOGEHYDJsQYRwJzqvRcvqoSBx6Yiq6uXXMnkaT6E2Nq\nij/nHFhmmdxpVAzFWNnaGtgjhLArsDCweAjhFmB8CGHFGbYRP5zdE3Tu3Pnb9xsaGmhoaChCLBVq\n+rL19tunk+WXWy53IkmqH3feCZMnpx5a5TdkyBCGDBnSrOco6pytEMK2wCmNPVsXAh/HGLuEEE4D\nlo4xnj6Lx9izVaH+8Id0FpfnJkpSeUyenPpmPf+wcmU/rmemYmsZoDewKvA20CHG+MksHmOxVaE+\n/TSdm9ivH2y2We40klT7/vxn+OgjuOmm3Ek0O9mLrUJYbFW2nj3TluIzz8B8jsCVpJIZPToNmH7p\nJVjBS8oqlhPkVXQHHwwLLAA33JA7iSTVrhjh97+Hzp0ttGqRK1uaq5EjYeed0yXIXhkjScV3yy1w\n+eVpF8GRO5XNbUSVzAknwNdfOw5Ckortk09g/fWhb1/7Y6uBxZZK5tNP0xUyd90FW22VO40k1Y4T\nT4QpU+C663In0byw2FJJ3XEHnH8+DB8O88+fO40kVb8RI2DXXeHll23TqBY2yKuk9t8fWrVKh6JK\nkppn2rTUFH/++RZatc6VLTXJ2LHpgOrnn4dVV82dRpKqV9euaZ7W4487WqeauI2osvjb39Ihqffc\nkzuJJFWn8eNhww1h0KD0VtXDYktl8eWX6YfDxRfD7rvnTiNJ1eegg9LuQJcuuZOoqSy2VDaDBqVD\nUkePhkUXzZ1GkqrHgw+mXq2XXoJFFsmdRk1lsaWy+s1v0m9mF1yQO4kkVYcvvoCf/hSuuQZ22SV3\nGhXCYkvk62uzAAAdT0lEQVRlZc+BJDXN6afD22/D7bfnTqJCWWyp7Lp1S7ennvKICUmak1GjYIcd\n0ttWrXKnUaGcs6WyO/LI1HNw1VW5k0hS5Zo2DY45Bs4910KrHrmypWZ77bV0hM/w4bD66rnTSFLl\nueYa6NULhg51pla1cxtR2Zx3HjzxBNx3H4QmfQtKUm0bNw5+9jMYMgR+8pPcadRcbiMqmz//Gd57\nL52fKElKYoTjjks3C6365cqWiuaZZ2CvvdLsmGWXzZ1GkvLr3TudujFiBCy4YO40Kga3EZVdx47w\n2WfQo0fuJJKU18cfp5la//oXbLFF7jQqFostZTdpUvrhcsMN6RJnSapXhxySVvkvvTR3EhVTIcVW\ny1KFUX1afHG49tp0ifOoUbDYYrkTSVL5PfAAPP44vPhi7iSqBK5sqSQOPRSWXBKuuCJ3EkkqL1f4\na5vbiKoYEyfCBhukIym22SZ3GkkqnxNOgP/9LxVbqj1uI6piLLNMGuJ35JHwwguebC+pPjz2GPTt\nm67KlqZzzpZKZs894Re/gLPPzp1Ekkrviy/gqKPgyithqaVyp1ElcRtRJfV//5e2E+++Ox3pI0m1\n6uST4cMP4bbbcidRKbmNqIqz3HLpt7wjjoDnn4eFF86dSJKKb+jQNMB01KjcSVSJ3EZUye27b1rd\n6tw5dxJJKr7PP0+/UF57radnaNbcRlRZfPghbLgh9OsHm2+eO40kFc+JJ6aTM3r2zJ1E5eA2oirW\nCiuk7cRDDknbiV6dKKkWDB6cjuNxeKnmxJUtldVBB8Hyy8Pll+dOIknNM2lSWrG/+mrYddfcaVQu\nDjVVxZs4Mf1wuvlmaNcudxpJKtxxx8HXXzu8tN64jaiKt8wy0K0bHH54umpnySVzJ5KkphswAO67\nz+1DzRtXtpTFscfCV19Bjx65k0hS00xfoe/RA3bcMXcalZvbiKoakyfDz34Gl14Ke+yRO40kzZsY\n4YAD4Ec/gssuy51GORRSbDlnS1ksthjcdFNa4froo9xpJGne9OqVzj38xz9yJ1E1cWVLWZ16Krzx\nBtx1F4Qm/Z4gSeX1zjuw6abw0EOw0Ua50ygXV7ZUdc45B15/Pa1ySVKlmjYNDj00nX9ooaWmcmVL\n2Y0eDQ0N8OSTsPbaudNI0g9dfHEaXvroo9CiRe40yskGeVWtK6+EW26BJ56A+efPnUaSvjNqFGy/\nPQwbBmuskTuNcnMbUVXrhBPSAa5/+1vuJJL0nS+/hN/+Fi680EJLhXNlSxVj/PjUC3HnnbDNNrnT\nSBJ07Ajvvw99+ngRjxInyKuqtWqVpssfcgiMHAlLLZU7kaR61r8/9O2bfh5ZaKk5XNlSxTn++DSh\nuVcvf8BJyuODD9JKe58+0LZt7jSqJPZsqSZcdBG88EJqmJekcps2DQ4+OA1dttBSMbiypYo0/eqf\nxx+HddbJnUZSPenSJW0hDh4MLW220Uwc/aCacu21cP318PTTsNBCudNIqgfDhsGvfgXPPQerrZY7\njSqRxZZqSoyw336pcf6qq3KnkVTrPvss9Wl16QL77ps7jSqVxZZqziefpB9+F18Me++dO42kWhVj\nmqe16KLQtWvuNKpkjn5QzVlqKbjjDth9d9h4Y2jdOnciSbWoW7fUK/rMM7mTqBa5sqWq8M9/wt13\nw9ChHucjqbhGjoQdd4THHoN1182dRpXO0Q+qWX/8Iyy9NPzlL7mTSKoln32WekMvv9xCS6Xjypaq\nxkcfpa3E666D3XbLnUZStYsRDjgg/SJ33XW506ha2LOlmrb88ql/a++9U1+F/VuSmuPaa+G11+Cp\np3InUa1zZUtV59JL4bbb0sBT529JKsTw4dC+PTz5JKy1Vu40qiaOflBdiBE6dIBll3XpX1LTffIJ\nbLIJXHBB6teSmsIGedWFEOCGG9JRGp6fKKkppp97uOuuFloqH3u2VJWWWALuugvatYOf/xw22CB3\nIknV4LzzYOLENEpGKhdXtlS1NtgALrkkHavx2We500iqdA88kFoP+vSBBRbInUb1xJ4tVb1jj01j\nIe66K20xStLM3nwTttwy/Zxo2zZ3GlUze7ZUly6/HMaNg/PPz51EUiX64gvYZx8480wLLeXhypZq\nwrhxsNlmcP31DjyV9J0Y4dBDYepUuPVWV7/VfA41Vd1aaSXo3Rv22ivN3/rxj3MnklQJrrkmnX34\n1FMWWsrHlS3VlK5d4bLL4Omn0xWLkurXo4+mmXxPPOHgUhWPQ00l4He/gw8/TJd2z2dXolSX/vOf\n1BB/yy2w446506iW2CAvAVdckYqtc8/NnURSDpMnw557whlnWGipMjS72AohrBJCeCSEMDqE8GII\n4aTG+5cOIQwIIbwaQngohLBk8+NKc7fgguny7m7doG/f3GkkldO0aXDIIfCLX8BJJ+VOIyXN3kYM\nIbQCWsUYR4YQFgOGA3sChwMfxxgvDCGcBiwdYzx9Fo93G1El8eyz6UiOgQPTlHlJta9z5/R3/pFH\n0i9eUrFl2UaMMY6PMY5sfH8yMAZYhVRw9Wz8sp7AXs19LakpfvELuPpq2GMP+OCD3Gkkldpdd0GP\nHnDPPRZaqixFHf0QQmgN/Bx4GlgxxjgBUkEWQlihmK8lzYsOHeCVV9JIiCFDYOGFcyeSVAojR8Jx\nx8FDD8GKK+ZOo1o06atJPPr2owU9tmhXIzZuIQ4B/h5j7BdCmBhjXGaGz38cY1x2Fo+LnTp1+vbj\nhoYGGhoaipJJgjTU8KCD0vu9ejlrR6o177+frjy8+GLYb7/caVQrpkydwjPvP0P3e7rzyOBHGD95\nPCsvsTJv9X0rz+iHEEJLoD/wQIzx8sb7xgANMcYJjX1dg2OM683isfZsqeT+9z/Ybjto3x5mqO0l\nVbnJk2GbbdIq9uk/6AqW5l2MkTH/N4aBbwxk0H8GMfTtoay59Jrs0GYHdmyzI1uvtjWLzL9Ivjlb\nIYSbgf+LMf5xhvu6ABNjjF1skFclGD8eNt8cLrwQ9t8/dxpJzTV1amoRWHHFdPWxq9ZqqnGTxjHo\nzUHf3hZosQA7ttmRHdfcke1ab8fyiy7/g8dkKbZCCFsDQ4EXgdh4OxMYBvQGVgXeBjrEGD+ZxeMt\ntlQ2I0emuTv9+sFWW+VOI6k5OnaE0aPhgQdg/vlzp1E1mPTVJIa8NSQVV/8ZxAeTPqDdGu3Ysc2O\n7NBmB9os3YYwl6rdCfLSPHjwQTjsMBg61DMUpWp15ZVw7bXw5JOw1FK506hSTe+7mr5yNXL8SDZf\nZfNvi6uNWm1Ei/laNOk5LbakeXTDDXD++elw2hW8TlaqKv37wzHHpDMP11gjdxpVknntu2oOiy2p\nCf7617TKNXgwLLpo7jSS5sVzz6ULXfr3Tz2YUiF9V81hsSU1QYxpO/GTT9IQxBZNW0mWVGavv56u\nPLz22nT2oepTMfqumsNiS2qir7+G3XZLvVtXXeXVTFKlmjAhXdRy2mlpC1H1oxR9V81hsSUV4NNP\noW3bNPjUOT1S5Zk0CRoa0tFbzsmrfeXou2oOiy2pQOPGwS9/CWeeCUcdlTuNpOm+/hp+9Sto0yZt\nH7r6XJvK3XfVHBZbUjOMHQvbbpu2E/feO3caSdOmwcEHwxdfpEOm7ausHbn7rprDYktqpuefh513\nhjvugHbtcqeR6leMcMopMGwYDBzoIfLVrtL6rprDYksqgkcfTYfZ3n8/bLpp7jRSfTrnHLj7bhgy\nBJZeOncaNVWl9101h8WWVCT33gu/+12awbXuurnTSPXl0kvhuuvSKQ8rrpg7jeZVNfVdNUchxVbL\nUoWRqtkee8B//5u2FIcOhdVXz51Iqg/du8Pll1toVYM59V112rZTRfddlZsrW9IcXHHFdz/4V145\ndxqptt15J/zxj2nrcO21c6fRzGqp76o53EaUSuDCC+HGG9M/AK1a5U4j1ab+/eHII1Mz/IYb5k4j\nqO2+q+aw2JJK5O9/T791DxkCyy2XO41UWx5+GA44wPMOK0G99F01h8WWVCIxwtlnpysUH3nEq6Ok\nYpleaN19dzr3UOVVzfOucrHYkkooRvjTn+Cxx2DQIFhiidyJpOr2yCOw//5pYOm22+ZOUx/su2o+\niy2pxGKEE05Iw08feACWXDJ3Iqk6DR4MHTpAnz7p3EOVhn1XxWexJZXBtGlw0knw7LPw4INuKUpN\nNWRIGhxsoVUa9l2VlsWWVCYxpkvUhw6FAQNg2WVzJ5Kqw6OPwr77Qu/esN12udPUBvuuystiSyqj\nGOG001KxNXAgLO8vi9IcDRwIBx2Uruz17NHC2XeVl8WWVGYxwl/+An37pquqnHgtzVq/fnD00XDP\nPfDLX+ZOU13su6osFltSJuecA7ffngqulVbKnUaqLLffDiefDPfdB5tskjtNdbDvqnJZbEkZdekC\n11+fthXXWit3GqkydO8OnTrBQw/BT3+aO03lsu+qelhsSZl17Qp/+1safvqzn+VOI+V12WXpNnCg\nZx3OzL6r6mWxJVWAPn3g+OPTROy2bXOnkcovxrS1fuutaWt9tdVyJ8rPvqvaYbElVYgBA+A3v4Gb\nboLddsudRiqfqVPTLxvDhqUV3no+vH3mvqsFWy747cqVfVfVy2JLqiBPPw177gkXXwy//W3uNFLp\n/e9/abTD5MnpqsPFF8+dqLxm7rsaP3k87dZoxw5r7GDfVQ2x2JIqzOjR0L59+k3/1FPBn7OqVRMn\nwh57wOqrQ48esMACuROV3sx9Vy9MeIHNV96cHdrsYN9VDbPYkirQe++lrcQtt4SrroKWLXMnkorr\n3Xdhl13SLxYXXgjzzZc7UWnMru9q+tagfVf1wWJLqlCffZbOgmvRIk3PrrftFdWukSPTilbHjnDK\nKbnTFN+c+q7ardGO5RZZLndElZnFllTBpkyB446DESOgf3+Hn6r63XsvHHkkXH01dOiQO01x2Hel\nubHYkipcjHD++WkeV//+sMEGuRNJTRcjXHJJuv3rX7DZZrkTFc6+KzWVxZZUJXr1Stsu3bunKxal\navH11+mCj2efTStb1TZDy74rNZfFllRFhg2DvfeGY4+Fs87ySkVVvokTYd99YbHF0i8Miy2WO9G8\nse9KxWSxJVWZceNSwbXaauly+UUXzZ1ImrUXX0zfq3vumc4BbVHBO2v2XamULLakKvTll2l1a+RI\n6NcvzSmSKskdd8CJJ8Kll1bmgF77rlROFltSlYoxHdh74YXpPLntt8+dSIJvvoHTT0/T4O+5B37+\n89yJEvuulJPFllTlHn44rRwcfzyceWbtDodU5fvwQ9h/f1hwwdSftcwyefPYd6VKYbEl1YD334cD\nDkiDT2+5BZZdNnci1Zunn06F1sEHw9/+lqc/y74rVSqLLalGTJmSVrZ69063zTfPnUj1YNo0uOii\nND+ra9fyjiWx70rVwmJLqjH9+sHRR6fRECed5HgIlc6HH8Ihh8DkyWnbsNTzs+y7UrWy2JJq0Jtv\nwoEHpp6ZHj2gVavciVRrHn44FVqHHZa2DUt1WLp9V6oFFltSjZoyBf7+d+jWLW3v7L577kSqBVOm\npOLqxhvh5pthhx2K+/z2XakWWWxJNe7xx1PTcvv28M9/wiLusqhAL72UVrNatUorpiuu2PzntO9K\n9cBiS6oDn36aRkMMH56uVtx009yJVE2mTk0N8BdeCP/4Bxx5ZOG9gPZdqR5ZbEl15Pbb4Q9/gMMP\nh86dYaGFcidSpXvjjdSXNd98cNNNsMYaTX8O+65U7yy2pDozYQKccEI6t+7GG2GrrXInUiWaOhWu\nvTYV5WedBR07zvvAXPuupO+z2JLq1F13pdEQ++8P557rgdb6zqhRcMwxMP/86eKK9dab89fbdyXN\nmcWWVMc+/jhtKz75JFx5Jey6a+5EyumLL+Ccc9KK53nnpd6sWa1m2XclNY3FliQeeihtLf7kJ+lw\n69atcydSuQ0YAMcdB5ttBpde+sPZbPZdSYWz2JIEwFdfpWNXLr0U/vhH+NOf0oHCqm1vvgl//jOM\nGAHXXJNGhIB9V1IxWWxJ+p633kpbi6NHw+WXp398/Te19kyalMY4dO0KJ58MJ3acwqiJ9l1JpWCx\nJWmW7rsvrW6ttFJa8dp449yJVAzTpqVZa2ecGdl0lzFs0mEgz02070oqJYstSbP1zTfQvXs6nmX7\n7VPT9Oqr506lQsQId9w3jjO7D2LyCoOgzSAWW9i+K6kcLLYkzdWkSemon6uugiOOgNNOg+X8d7ni\nTe+7uvmJQdz/yiC+bDmezZZrxyFtd2CnNe27ksrFYkvSPPvgg7TK1acPHH00nHIKLL987lSabuZ5\nV8+Pe4GFJ27O1LE7cNKvduDMwzdiwQXsu5LKzWJLUpO9805qrr7zTjjqqNTbtcIKuVPVn9nNu/rJ\nQjvyxqAdeHPI1px92iIcfbRXlko5WWxJKti770KXLunMxUMPTUe62NNVWrObd7X9GjvAf9px7cXL\n8dZbaZzD4YfDIva4S9lZbElqtvffT/O5evRIjfQnnwxbbpk7VW2Y27yrVRZtwz33BLp0SecZnn46\ndOiQjtqRVBkstiQVzaRJ6aiXyy9P24onnwx77+0//E0xr+cMvvdempHVrRuss07ayt1tN2eiSZXI\nYktS0U2dCvfem1a7xo5NW4xHHglrr507WeVpyjmDMcLgwXD11entQQelI3Z+8pPMfwhJc2SxJamk\nxoyBG25IgzTXXTc11O+zT333EjX1nMHXX4fbboNbb4WFFoLf/x5++1tYfPFMfwBJTWKxJaksvv4a\n/v3vVHg9/TT86lept2jHHWv/SrlCzhn8+ON0tectt6TzCw84AA4+GDbZxK1CqdpYbEkqu3Hj4O67\n07yul16C3Xf/rvBaYIHc6ZpvXvuuZvbuu9CvX7oNGwa77poKrB13tO9NqmYWW5Kyev/9VHj17g0v\nvggNDbDLLum2xhq5082bpvRdzWjaNBg5Mq349euX5pftthvsuSfstBMstliGP4ykoqvIYiuEsAtw\nGTAfcEOMsctMn7fYkmrQRx/BwIHw4IPw0EOw1FKp6Np2W9hqK2jVKnfC7zS17wpScTV6dGpuHzwY\nhg5NV222b58KrK23hpYtM/xhJJVUxRVbIYT5gNeA7YFxwLPAATHGV2b4GostqcZNmwYvvJCKrsce\ng6eeSsXX1lunwmuLLWD99cvX71VI39X48TB8eLo991zqVVtiCWjXDrbbLq3i/ehH5ckvKZ9KLLa2\nADrFGNs3fnw6EGdc3bLYkurPtGnw6qvwxBPw5JOpp+mNN6B1a9hgA/jpT9PbtddOU+ybe6XevPZd\nxZjOjBw7Fl57Lb195RUYMQK++AI23TQ1tW+6KWy2Gay6alH+c0iqIpVYbO0D7BxjPKbx498Cm8UY\nT5rhayy2JPHVV6kAe+ml1O/14oupAHvnnbTitfrqsNpq6bbssrD00rDMMunt0kunnqiWLaFFC2jR\nIvKfyWN4cvxAhr43iGc/HMpKC6/JRkvsyPoL78Aq07bmvx8uwvjxfHv74AN4+21YeGH48Y/Tbe21\n09uNN06FoFcOSqraYotDD/3uQT//ebpJkiTlNnJkuk3Xs2fFFVtbAJ1jjLs0fuw2oqSiKKTvSpKa\nqxJXtloAr5Ia5D8AhgEHxhjHzPA1FluS5qrQeVeSVEwVV2zBt6MfLue70Q8XzPR5iy1JP1DovCtJ\nKqWKLLbmGsBiS1KjQuZdSVI5WWxJqir2XUmqNhZbkiqafVeSqp3FlqSKYt+VpFpjsSUpO/uuJNUy\niy1JZWfflaR6YrElqeTsu5JUzyy2JBWdfVeS9B2LLUlFYd+VJM2axZakgth3JUnzxmJL0jyx70qS\nCmOxJWmW7LuSpOKw2JL0LfuuJKn4LLakOmbflSSVnsWWVEfsu5Kk8rPYkmqYfVeSlJ/FllRj7LuS\npMpisSVVOfuuJKmyWWxJVca+K0mqLhZbUoWz70qSqpvFllSB7LuSpNphsSVVgElfTeLRtx/9dvXK\nvitJqh0WW1IGU6ZOYdj7wxj45kD7riSpxllsSWUwve9q0JuDGPjmQPuuJKmOWGxJJTJu0jgefvPh\nb1ev7LuSpPpksSUViX1XkqRZsdiSCmTflSRpXlhsSfPIvitJUiEstqQ5sO9KktRcFlvSDOy7kiQV\nm8WW6pp9V5KkUrPYUl2x70qSVG4WW6p59l1JknKy2FLNse9KklRJLLZU9ey7kiRVMostVR37riRJ\n1cRiS1XBvitJUrWy2FJFsu9KklQrLLZUEey7kiTVKostZWHflSSpXlhsqWzsu5Ik1SOLLZWMfVeS\nJFlsqYjsu5Ik6YcstlQw+64kSZo7iy01iX1XkiQ1jcWW5si+K0mSmsdiS99j35UkScVlsVXn7LuS\nJKm0LLbqkH1XkiSVj8VWHbDvSpKkfCy2apB9V5IkVQ6LrRpg35UkSZXLYqtK2XclSVJ1sNiqEvZd\nSZJUnSy2KpR9V5Ik1QaLrQph35UkSbXJYisj+64kSap9FltlZN+VJEn1x2KrhOy7kiRJFltFZN+V\nJEmamcVWM9l3JUmS5sRiq4nsu5IkSU1hsTUX9l1JkqTmsNiaiX1XkiSpmCy2sO9KkiSVTl0WW/Zd\nSZKkcqmLYsu+K0mSlEvZi60QwoXA7sBXwBvA4THGzxo/dwZwBPAN0DHGOGA2zzHHYsu+K81oyJAh\nNDQ05I6hKuD3iprC7xfNq0KKrZbNfM0BwOkxxmkhhAuAM4AzQgjrAx2A9YBVgEEhhLXndQlrdn1X\nB294MD327GHfVR3zB6Lmld8ragq/X1RKzSq2YoyDZvjwaWCfxvf3AO6IMX4DvBVCGAtsBjwzq+eZ\nU99Vp2072XclSZKqVnNXtmZ0BHB74/srA0/N8Ln3G++bpZUuWenbvquee/W070qSJNWMufZshRAG\nAivOeBcQgbNijP9u/JqzgI1jjPs0fnwl8FSMsVfjx92B+2OM98zi+SvjrB5JkqR5UPSerRjjjnP6\nfAjhMGBXoN0Md78PrDrDx6s03jer53d/UJIk1az5mvPgEMIuwJ+BPWKMX83wqXuBA0IIC4QQ1gDW\nAoY157UkSZKqUXN7tq4EFgAGNjawPx1j/H2M8eUQQm/gZWAK8Ptsp01LkiRllH2oqSRJUi1r1jZi\nc4QQ9g0hvBRCmBpC2Himz50RQhgbQhgTQtgpV0ZVphBCpxDCeyGEEY23XXJnUmUJIewSQnglhPBa\nCOG03HlUuUIIb4UQXgghPB9CsN1F3xNCuCGEMCGEMGqG+5YOIQwIIbwaQngohLDk3J4nW7EFvAj8\nGnh0xjtDCOvx3UDU9sA1wSFb+qFLYowbN94ezB1GlSOEMB9wFbAz8BPgwBDCunlTqYJNAxpijBvF\nGDfLHUYVpwfpZ8mMTgcGxRjXAR4hDXSfo2zFVozx1RjjWNIoiRntSeNA1BjjW8D0gajSjCzANTub\nAWNjjG/HGKcAd5B+rkizEsi78KAKFmN8HPjvTHfvCfRsfL8nsNfcnqcSv8FWBt6d4eM5DkRV3Toh\nhDAyhNB9XpZwVVdm/hnyHv4M0exF0kVez4YQjs4dRlVhhRjjBIAY43hghbk9oJgT5H9gXgaiSrMy\np+8d4BrgnBhjDCGcC1wCHFn+lJJqwNYxxg9CCMuTiq4xjasZ0rya65WGJS225jYQdTbmeSCqalcT\nvne6ARbumtH7wGozfOzPEM1WjPGDxrcfhRD+RdqGttjSnEwIIawYY5wQQmgFfDi3B1TKNuKM/TcO\nRNUcNX5zT7c38FKuLKpIzwJrhRBWDyEsABxA+rkifU8IYZEQwmKN7y8K7IQ/T/RDgR/WKYc1vn8o\n0G9uT1DSla05CSHsRRqKuhzQP4QwMsbY3oGomgcXhhB+TrqK6C3gd3njqJLEGKeGEE4ABpB+obwh\nxjgmcyxVphWBfzWe0dsSuC3GOCBzJlWQEEIvoAFYNoTwDtAJuADoE0I4AnibNEFhzs9jHSNJklQ6\nlbKNKEmSVJMstiRJkkrIYkuSJKmELLYkSZJKyGJLkiSphCy2JEmSSshiS5IkqYT+HxR0naSkqxwc\nAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x7fafeeccb450>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "import matplotlib.pyplot as plt\n",
    "%matplotlib inline\n",
    "plt.rcParams['figure.figsize'] = (10.0, 8.0) # set default size of plots\n",
    "plt.rcParams['image.interpolation'] = 'nearest'\n",
    "plt.rcParams['image.cmap'] = 'gray'\n",
    "\n",
    "x = np.linspace(-10, 10, 200)\n",
    "# plt.plot only takes ndarray as input. Explicitly convert MinPy Array into ndarray.\n",
    "plt.plot(x.asnumpy(), foo(x).asnumpy(),\n",
    "         x.asnumpy(), d_foo(x).asnumpy(),\n",
    "         x.asnumpy(), d_2_foo(x).asnumpy(),\n",
    "         x.asnumpy(), d_3_foo(x).asnumpy())\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Just as you expected."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Autograd also differentiates vector inputs. For example:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[ 2.  4.  6.  8.]"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "x = np.array([1, 2, 3, 4])\n",
    "d_foo(x)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Gradient of Multivariate Functions\n",
    "As for multivariate functions, you also need to specify arguments for derivative calculation. Only the specified argument will be calcualted. Just pass the position of the target argument (of a list of arguments) in `grad`. For example:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def bar(a, b, c):\n",
    "    return 3*a + b**2 - c"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We get their gradients by specifying their argument position."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[3.0, 6.0, -1.0]\n"
     ]
    }
   ],
   "source": [
    "gradient = grad(bar, [0, 1, 2])\n",
    "grad_array = gradient(2, 3, 4)\n",
    "print grad_array"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`grad_array[0]`, `grad_array[1]`, and `grad_array[2]` are gradients of argument `a`, `b`, and `c`.\n",
    "\n",
    "The following section will introduce a more comprehensive example on matrix calculus."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Autograd for Loss Function\n",
    "\n",
    "Since in world of machine learning we optimize a scalar loss, Autograd is particular useful to obtain the gradient of input parameters for next updates. For example, we define an affine layer, relu layer, and a softmax loss. Before dive into this section, please see [Logistic regression tutorial](../get-started/logistic_regression.rst) first for a simpler application of Autograd."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def affine(x, w, b):\n",
    "    \"\"\"\n",
    "    Computes the forward pass for an affine (fully-connected) layer.\n",
    "    The input x has shape (N, d_1, ..., d_k) and contains a minibatch of N\n",
    "    examples, where each example x[i] has shape (d_1, ..., d_k). We will\n",
    "    reshape each input into a vector of dimension D = d_1 * ... * d_k, and\n",
    "    then transform it to an output vector of dimension M.\n",
    "    Inputs:\n",
    "    - x: A numpy array containing input data, of shape (N, d_1, ..., d_k)\n",
    "    - w: A numpy array of weights, of shape (D, M)\n",
    "    - b: A numpy array of biases, of shape (M,)\n",
    "    Returns a tuple of:\n",
    "    - out: output, of shape (N, M)\n",
    "    \"\"\"\n",
    "    out = np.dot(x, w) + b\n",
    "    return out\n",
    "\n",
    "def relu(x):\n",
    "    \"\"\"\n",
    "    Computes the forward pass for a layer of rectified linear units (ReLUs).\n",
    "    Input:\n",
    "    - x: Inputs, of any shape\n",
    "    Returns a tuple of:\n",
    "    - out: Output, of the same shape as x\n",
    "    \"\"\"\n",
    "    out = np.maximum(0, x)\n",
    "    return out\n",
    "\n",
    "def softmax_loss(x, y):\n",
    "    \"\"\"\n",
    "    Computes the loss for softmax classification.\n",
    "    Inputs:\n",
    "    - x: Input data, of shape (N, C) where x[i, j] is the score for the jth class\n",
    "    for the ith input.\n",
    "    - y: Vector of labels, of shape (N,) where y[i] is the label for x[i] and\n",
    "    0 <= y[i] < C\n",
    "    Returns a tuple of:\n",
    "    - loss: Scalar giving the loss\n",
    "    \"\"\"\n",
    "    N = x.shape[0]\n",
    "    probs = np.exp(x - np.max(x, axis=1, keepdims=True))\n",
    "    probs = probs / np.sum(probs, axis=1, keepdims=True)\n",
    "    loss = -np.sum(np.log(probs) * y) / N\n",
    "    return loss"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Then we use these layers to define a single layer fully-connected network, with a softmax output."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "class SimpleNet(object):\n",
    "    def __init__(self, input_size=100, num_class=3):\n",
    "        # Define model parameters.\n",
    "        self.params = {}\n",
    "        self.params['w'] = np.random.randn(input_size, num_class) * 0.01\n",
    "        self.params['b'] = np.zeros((1, 1))  # don't use int(1) (int cannot track gradient info)\n",
    "\n",
    "    def forward(self, X):\n",
    "        # First affine layer (fully-connected layer).\n",
    "        y1 = affine(X, self.params['w'], self.params['b'])\n",
    "        # ReLU activation.\n",
    "        y2 = relu(y1)\n",
    "        return y2\n",
    "\n",
    "    def loss(self, X, y):\n",
    "        # Compute softmax loss between the output and the label.\n",
    "        return softmax_loss(self.forward(X), y)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We define some hyperparameters."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "batch_size = 100\n",
    "input_size = 50\n",
    "num_class = 3"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here is the net and data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "net = SimpleNet(input_size, num_class)\n",
    "x = np.random.randn(batch_size, hidden_size)\n",
    "idx = np.random.randint(0, 3, size=batch_size)\n",
    "y = np.zeros((batch_size, num_class))\n",
    "y[np.arange(batch_size), idx] = 1"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now get gradients."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "gradient = grad(net.loss)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Then we can get gradient by simply call `gradient(X, y)`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "d_x = gradient(x, y)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Ok, Ok, I know you are not interested in `x`'s gradient. I will show you how to get the gradient of the parameters. First, you need to define a function with the parameters as the arguments for Autograd to process. Autograd can only track the gradients **in the parameter list**."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def loss_func(w, b, X, y):\n",
    "    net.params['w'] = w\n",
    "    net.params['b'] = b\n",
    "    return net.loss(X, y)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Yes, you just need to provide an entry in the new function's parameter list for `w` and `b` and that's it! Now let's try to derive its gradient."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# 0, 1 are the positions of w, b in the paramter list.\n",
    "gradient = grad(loss_func, [0, 1])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note that you need to specify a list for the parameters that you want their gradients.\n",
    "\n",
    "Now we have"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "d_w, d_b = gradient(net.params['w'], net.params['b'], x, y)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "With `d_w` and `d_b` in hand, training `net` is just a piece of cake."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Less Calculation: Get Forward Pass and Backward Pass Simultaneously\n",
    "Since gradient calculation in MinPy needs forward pass information, if you need the forward result and the gradient calculation at the same time, please use `grad_and_loss` to get them simultaneously. In fact, `grad` is just a wrapper of `grad_and_loss`. For example, we can get"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "from minpy.core import grad_and_loss\n",
    "forward_backward = grad_and_loss(bar, [0, 1, 2])\n",
    "grad_array, result = forward_backward(2, 3, 4)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`grad_array` and `result` are result of gradient and forward pass respectively."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python [Root]",
   "language": "python",
   "name": "Python [Root]"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 2
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython2",
   "version": "2.7.12"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}
